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Background 


This  was  a  request  for  acquisition  of  a  high-end  hybrid  CPU-GPU  computing  facility, 
designed  specifically  to  dramatically  enhance  current  computing  capabilities,  and 
support  algorithm  development  for  computationally  intensive  stochastic  and  high- 
resolution  simulations.  The  system  will  be  used  to  support  research  in  the  areas  of 
interest  to  a  new  MURI  AFSOR. 

The  new  facility  is  available  to  9  Pis,  14  PhD  and  6  post-doc  students  who  work  on  the 
projects  supported  by  DoD  from  Brown  and  also  from  other  academic  institutions, 
partners  in  the  AFSOR  MURI  Project. 

The  computational  platform  comprises  of  integrated  CPU  and  GPU  systems;  specifically, 
IBM  Dual-processor/quad-core  nodes  with  48GB  RAM  each  and  40Gb/s  QDR  Infiniband, 
and  nVIDIA  Tesla  GPU  system  with  a  total  of  88  GPUs.  With  the  addition  of  the  two 
requested  18-port  switch  leaf-modules,  the  44  host  computers  will  be  connected  to  a 
Qlogic  12800  switch  already  installed  at  Brown.  In  addition,  66TB  (usable)  disk  is 
requested,  which  will  be  integrated  with  the  GPFS  filesystem  already  installed  at  Brown. 
The  request  also  included  a  number  of  small  desktop  systems  to  allow  researchers  to 
access  the  facility  and  effectively  visualize  the  computational  results.  This  latter 
component  was  not  executed  due  to  unexpected  scheduling  issues  and  the  associated 
funds  were  returned  to  AFOSR. 

Funding  through  DURIP  has  been  instrumental  for  purchasing  and  establishing  the 
proposed  facility  for  which  no  other  funds  would  have  been  available.  Establishing  the 
facility  has  helped  to  pave  the  way  to  greatly  enhance  the  ability  to  locally  develop  and 
test  novel  parallel  algorithms  for  solving  large  stochastic  systems  arising  in  the  areas  of 
DoD  interest.  It  is  widely  acknowledged  that  such  simulations  tax  the  computational 
facilities  much  heavier  than  classical  deterministic  modeling  approach.  We  also  make 
use  of  the  equipment  for  postprocessing,  e.g.,  to  visualize  very  large  temporal  datasets 
resulting  from  the  simulation  transient  phenomena  in  general  complex  geometries.  In 
addition,  it  will  also  significantly  enhance  educational  and  outreach  programs. 

The  central  components  of  the  facility,  i.e.,  the  GPU/CPU  hybrid  cluster  and  the  external 
storage  systems  are  fully  integrated  with  the  networks  and  storage  facilities  of  Brown's 
Center  for  Computation  and  Visualization  (CCV)  to  further  enhance  the  use  of  the  facility 
and  leverage  the  purchase  against  existing  infrastruture  and  system  support. 

The  central  components  of  the  facility  are  all  highly  flexible  and  expandable  in  capacity. 
Combined  with  the  upgrade  of  the  existing  network  hierarchy,  central  in  order  to 
achieve  the  expected  performance  locally  as  well  as  improve  the  connectivity  to  DOD 
HPC  sites,  the  facility  paved  the  way  for  an  extensive,  efficient,  and  reliable  use  of  DOD 
HPC  resources.  The  acquisition  has  established  a  powerful  computational  platform  for 
novel  algorithm  development  and  research-related  education  in  areas  of  research 


relevant  to  the  missions  of  DOD  and  will,  due  to  the  emphasis  in  the  design  of  the 
facility  on  flexibility,  expandability,  and  connectivity  to  existing  resources,  have  an 
estimated  useful  life  of  3-5  years. 

The  proposed  system  has  been  purchased,  is  fully  functions  and  heavily  utilized  by 
faculty  and  students  involved  in  AFOSR  related  research. 

Equipment  purchased 

From  IBM  -  44  IBM  iDataplex  dx360  M3  nodes,  each  with: 

■  2  Intel  Xeon  E5630  processors  (4-core,  2.53GHz) 

•  24GB  DDR3  RAM 

•  2  nVIDIA  Tesla  M2050  GPUs  (Fermi  architecture,  3GB  GPU  memory) 

•  Mellanox  QDR  (40Gb/s)  Infiniband  HCA 

and  including  associated  rack  and  Ethernet  management  switch  hardware. 

From  IBM  -  3  Qlogic  Infiniband  switch  modules,  installed  in  the  extant  Qlogic  12800-180 
QDR  Infiniband  switch.  These  connections  provide  non-blocking  access  to  ~300 
terabytes  of  IBM  GPFS  parallel  filesystem  storage  for  data  and  run-time  scratch  storage. 

From  Server  Supply  -  33  2TB  disk  drives,  1  dual  core  Xeon  server  and  1 146GB  SAS  driver 
as  disk  server. 


